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Executive  Summary 


The  Spiral- 1  Maritime  Domain  Awareness  (MDA)  Assessment  Team  of  the  Naval  Postgraduate 
Sehool  (NPS)  eondueted  a  Teehnieal  Risk  Reduetion  Limited  Objeetive  Experiment  (TRRLOE) 
from  3  through  5  June  2008.  The  event  took  plaee  in  San  Diego,  California,  at  the  Spaee  and 
Naval  Warfare  Systems  Center,  (SPAWAR-SD),  Building  606,  Eab  140.  The  goals  of  this  event 
were  twofold. 

1 .  Identify  and  reeommend  mitigation  to  risks  to  the  sueeessful  exeeution  of  the 
EAIRGAME  event  seheduled  for  the  week  of  14  July  2008. 

2.  Continue  to  eolleet  data  to  further  assess  MDA  Spiral- 1  teehnologies. 

Our  findings  from  the  TRRLOE  indieate  that  several,  signifieant  risks  to  EAIRGAME  exeeution 
exist.  Eor  example: 

•  Database  differenees  between  the  several  operational  nodes  may  make  it  diffieult  to 
identify  ships  for  eollaboration  and  handoff. 

•  Databases  are  routinely  re-based  with  revised  software  versions  making  historieal  traeks 
less  useful  for  EAIRGAME  exereises. 

•  Comprehensive  Maritime  Awareness  (CMA)  user  aeeounts  must  inelude  “alert 
permissions”  in  order  to  manipulate  and  exeeute  CMA  alerts. 

•  Operators  will  be  unfamiliar  with  how  to  use  the  suite  of  Spiral- 1  systems  in  eonjunetion 
with  the  systems  already  in  use.  Training  should  inelude  how  to  exeeute  the  planned 
EAIRGAME  events. 

•  The  events  were  CMA-eentrie  and  did  not  suffieiently  exereise  the  speeifie  eapabilities 
provided  by  the  other  systems. 

TRRLOE  provided  limited  assessment  of  MDA  Spiral- 1  systems  and  their  use.  It  was 
determined  that 

•  Operators  were  able  to  eomplete  assigned  Requests  for  Information  (REIs)  in  times  that 
were  signifieantly  shorter  than  normally  required  to  obtain  the  types  of  information 
requested.  This  was  a  limited  and  perhaps  non-representative  sample  size. 

o  1  min  to  determine  when  ship  was  built  after  White  Cell  request, 
o  2  min  to  identify  eargo  on  a  named  ship, 
o  6  min  to  loeate  a  ship,  details  on  type  and  erew. 
o  7  min  to  determine  person-of-interest  was  not  on  reported  ship. 

•  Only  limited  training  on  the  systems  was  provided,  yet  produeed  a  reasonable  degree  of 
operator  eompetenee.  Indieations  are  that  the  systems  are  fairly  easy  to  learn  and  use. 
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•  CMA  and  FASTC2AP  have  some  human-system  interaction  difficulties  with  the  user 
interface. 

•  FASTC2AP  alerts  are  difficult  to  configure  and  non-intuitive  in  terms  of  the  results  that 
will  be  produced.  Some  redesign  is  needed,  preferably  with  consultation  with  watch- 
floor  SMEs  to  determine  what  they  need. 

In  summary,  the  effort  revealed  relevant  and  useful  information  concerning  the  risks  and 
assessments  of  Spiral- 1  technologies.  More  detailed  risk  and  assessment  identifications  are 
discussed  and  analyzed  within  this  report. 


2 


1.0  TRRLOE  Background 


The  Technical  Risk  Reduction  Limited  Objective  Experiment  was  conducted  3-5  Jun,  2008,  at 
SPAWAR,  San  Diego,  California,  Lab  140. 

The  purpose  of  the  TRRLOE  was  twofold; 

Primary  Purpose:  To  identify  and  thereby  reduce  the  risk  to  the  conduct  of  the 
EAIRGAME  test  event  of  Maritime  Domain  Awareness  (MDA)  Spiral- 1  technologies,  and 
Secondary  Purpose:  To  provide  information  for  MDA  assessment. 

EAIRGAME  plans  to  test  MDA  Spiral- 1  technologies  in  a  distributed  operational  environment 
that  incorporates  several  operational  nodes  (PACELT,  NAVCENT,  MIECPAC,  etc). 

The  TRRLOE  laboratory  event  was  run  in  two  rooms.  One  room  contained  the  White  Cell 
which  directed  test  activities.  The  second  room  contained  three  operators.  They  were  situated  so 
that  they  could  either  collaborate  or  operate  independently.  Communications  between  the  White 
Cell  and  operators  was  via  instant  messenger. 

The  Spiral- 1  technologies  are  listed  below  with  no  explanation  of  their  purpose. 

•  CMA 

•  MAGNET 

•  EASTC2AP 

•  Data  Sharing  COI 

•  Google  Earth 

•  Tripwire 

•  Global  Trader 

•  LiNX 

•  E-MIO  Wireless 

TRRLOE  tested  six  of  the  systems  in  a  laboratory  environment. 

•  CMA 

•  EASTC2AP 

•  Data  Sharing  COI 

•  Google -Earth 

As  much  as  possible,  TRRLOE  exercised  the  events  that  are  planned  for  EAIRGAME. 

TRRLOE  evaluations  focused  on; 

•  Evaluating  risk  factors  to  EAIRGAME  success  due  to 

o  Personnel  training 
o  System  performance 
o  Information  /  database  availability 
o  Test  conduct 

In  spite  of  event  structural  differences,  risk  information  was  obtained  for  all  of  these  categories. 


3 


This  report  contains: 

o  Descriptions  of  the  FAIRGAME  and  TRRLOE  events  and  reasons  for  differences 
o  TRREOE  test  plan 
o  TRRLOE  planned  test  conduct 
o  TRRLOE  executed  test  conduct 
o  EAIRGAME  risk  reduction  results 
o  MDA  assessment  results 
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2.0  Test  Structure  and  Events 


As  a  risk  reduction  test  for  FAIRGAME  (FG),  TRRLOE  executed  the  FG  events.  This  section 
lists  those  events  as  described  in  the  9  May,  Rev  0.6,  of  the  test  SOP  for  FG.  This  is  followed  by 
descriptions  of  the  TRRLOE  events  and  reasons  for  the  differences.  Differences  in  structure 
were  due  to  what  was  available  for  TRRLOE  in  the  laboratory. 


2,1  FAIRGAME  Events 

These  events  are  from  FAIRGAME  08-02;  MDA  Spiral  One  Test  SOP,  Rev  .06,  9  May  2008. 

EVOl:  OPS  Tipper  RFI 

Description  A  Request  for  Information  (RFI)  is  received  by  the  operations  watch  from  a 
supported  command  concerning  a  vessel.  The  watch  officer  will  process  the  RFI  (reject,  answer 
locally,  forward)  by  performing  local  analysis. 

Activities  The  operations  watch  officer  will  process  the  RFI  (reject,  answer  locally,  forward) 
by  performing  local  analysis  and  then  either  closing  the  request  or  forwarding  the  RFI  to  the  next 
level  of  command  for  further  analysis. 

Inputs  The  RFIs  will  be  initiated  by  a  synthetic  message  from  the  white  cell  representing  a 
request  from  an  entity  normally  supported  by  the  watch  floor.  Because  each  watch  floor  has 
theater-specffic  responsibilities,  this  will  require  three  white  cell-generated  targets,  one  in  the 
NAVCENT  AOR,  one  in  the  PACFLT  AOR,  and  one  in  the  LANTFLT  AOR. 

Outputs  The  result  of  the  watch  floor  process  will  be  a  message  back  to  the  supported  activity 
(white  cell)  with  an  answer,  plus  potentially  a  message  or  data  request  from  the  watch  floor  to 
the  next  level  of  command  requesting  further  information. 

EV02:  Intelligence  Tipper  RFI 

Description  A  Request  for  Information  (RFI)  is  received  by  the  Intelligence  watch  from  a 
supported  command  concerning  a  vessel.  The  watch  officer  will  process  the  RFI  (reject,  answer 
locally,  forward)  by  performing  local  analysis. 

Activities  The  operations  watch  officer  will  process  the  RFI  (reject,  answer  locally,  forward) 
by  performing  local  analysis  and  then  either  closing  the  request  or  forwarding  the  RFI  to  the  next 
level  of  command  for  further  analysis. 

Inputs  The  RFIs  will  be  initiated  by  a  synthetic  message  from  the  white  cell  representing  a 
request  from  an  entity  normally  supported  by  the  watch  floor.  Because  each  watch  floor  has 
theater-specffic  responsibilities,  this  will  require  three  white  cell-generated  targets,  one  in  the 
NAVCENT  AOR,  one  in  the  PACFLT  AOR,  and  one  in  the  LANTFLT  AOR.  The  white  cell 
will  also  “seed”  the  TRIPWIRE  database  with  a  synthetic  report  implicating  the  vessel  provided 
as  a  target  to  NAVCENT. 

Outputs  The  result  of  the  watch  floor  process  will  be  a  message  back  to  the  supported  activity 
(white  cell)  with  an  answer,  plus  potentially  a  message  or  data  request  from  the  watch  floor  to 
the  next  level  of  command  requesting  further  information. 
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EV03:  AOR  Movement 

Description  The  activity  commander’s  direct  special  attention  be  PAID  by  their  watch  teams 
to  any  targets  moving  from  NAVCENT  to  PACFLT  AORs  based  on  classified  indicators. 
Activities  The  watch  team  should  look  for  vessels  transiting  on  or  near  the  chop  line  between 
AORs.  This  will  involve  manual  screening  of  vessels  as  well  as  setting  up  geographic  boundaries  for 
automatic  alarms.  It  will  also  initiate  coordination  and  collaboration  among  NAVCENT,  PACFLT, 
NMIC,  and  MIFCPAC. 

Inputs  The  white  cell  will  generate  a  “message”  from  the  activity  commander  directing  the 
watch  to  place  a  high  priority  on  any  targets  moving  between  NAVCENT  and  PACFLT  AORs. 
Outputs  Output  will  be  a  report  to  the  commander  of  any  vessels  in  or  near  the  chop  line  as 
well  as  a  collaboration  space  now  existing  among  the  test  activities. 

EV04:  VOI  Addition 

Description  NMIC  adds  three  new  named  vessels  to  the  high-interest  vessel  list  and  requests 
any  additional  information  available  at  the  theater  level.  One  vessel  is  in  the  NAVCENT  AOR, 
moving  toward  the  PACFLT  AOR,  two  others  are  nonexistent,  one  in  LANT  and  one  in  PAC. 
Activities  The  watch  conducts  analysis  of  the  new  additions.  Warning  is  promulgated  among 
the  players  based  on  location  and  movement.  Continuous  tracking  is  initiated,  and  responsibility 
passes  from  NAVCENT  to  PACFLT.  Nonexistent  vessels  trigger  horizontal  collaboration  and 
RFI  generation  to  next  level  of  command. 

Inputs  Existing  vessel  to  be  chosen  by  the  white  cell  based  on  an  existing  ship  with  the  proper 
geographic  location  and  movement.  A  message  is  generated  by  the  white  cell  to  the  NMIC  watch 
floor  who  should,  in  turn,  notify  the  appropriate  theater  watches. 

Outputs  The  output  is  a  state  change  in  which  the  new  targets  are  added  to  the  local  VOI  lists 
for  watch  floor  prosecution. 

EV05:  MTAC  Cueing 

Description  MTAC  report  indicates  absconder  debriefs  indicate  US  East  Coast  human 
trafficking  may  involve  three  named  ships. 

Activities  Collaboration  among  NMIC  and  MFICLANT  results  in  identification  and 
subsequent  tracking  of  merchant  ship  off  US  East  Coast. 

Inputs  White  cell  chooses  an  existing  vessel,  then  creates  synthetic  MTAC  report  that  is 
passed  to  the  ICC  and  MFICs. 

Outputs  The  vessel  is  added  to  VOI  lists  at  NMIC  and  at  MFICLANT. 

EV06:  Multivariate  Query 

Description  The  commander  asks  the  watch  for  a  list  of  all  vessels  in  a  specified  geographic 
area  with  certain  specified  attributes. 

Activities  The  watch  must  correlate  information  from  multiple  sources  to  answer  the  query. 
Inputs  White  cell  chooses  a  geographic  area  of  interest,  e.g.,  100  NM  around  Malacca  Strait 
for  NAVCENT,  around  Hawaii  for  PACFLT,  around  Straits  of  Juan  de  Fuca  for  MFIC  PAC,  etc. 
In  that  area,  the  watch  is  asked  to  report  all  ships  registered  in  one  of  three  specified  countries 
and  carrying  hazardous  cargo. 

Outputs  A  list  of  vessels  fitting  all  criteria. 
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EV07:  Cargo  Container  Query 

Description  A  standard  shipping  container  in  transit  is  suspected  of  carrying  something  other 
than  its  designated  eargo. 

Activities  The  wateh  is  asked  to  determine  container  history,  eurrent  loeation,  and  future  plan. 
Inputs  White  cell  provides  a  suitable  eontainer  as  a  target  for  this  activity. 

Outputs  A  shipping  history,  current  loeation,  and  shipping  plan.  Global  Trader  is  expected  to 
be  the  main  eontributing  teehnology  for  gathering  this  information. 

EV08:  Data  Provenance 

Description  The  commander  requests  all  available  data  on  a  target  ship  and  specified  that  the 
souree  of  eaeh  data  element  be  reported  with  the  data  itself 

Activities  The  wateh  is  asked  to  investigate  a  VOI  but  must  report  the  source  of  all  data. 
Inputs  White  eell  provides  a  target  suitable  to  each  participating  activity. 

Outputs  A  report  is  ereated  and  returned  by  the  watch  containing  all  relevant  information  as 
well  as  its  source. 

EV09:  Contingent  Geographic  Alarms 

Description  The  aetivity  commanders  request  immediate  notification  of  vessels  with  specified 
attributes  penetrating  certain  geographic  boundaries. 

Activities  The  watch  team  will  build  boxes  or  fences  around  the  target  areas  using  automated 
tools  sueh  as  CMA  or  FASTC2AP. 

Inputs  The  white  eell  will  provide  geographie  areas  of  interest  to  the  “commander” 
appropriate  to  each  activity. 

Outputs  Output  will  be  automated  alarms  in  plaee  and  operating.  The  completion  of  alarm 
box  building  is  the  event  eompletion,  not  the  triggering  of  the  alarm  by  a  real-world  ship. 


2,2  TRRLOE  /  FAIRGAME  Structural  Differences 

Differences  in  systems,  operational  units,  operators,  training,  and  databases  neeessitated  event 
modifications.  This  sub-section  lists  those  differenees  and  their  effect. 

Systems; 

1 .  FAIRGAME  will  have  the  full  suite  of  Spiral- 1  systems,  with  the  exception  of  EMIO. 
TRREOE  had  available:  CMA,  EASTC2AP,  Google  Earth,  and  Data  Sharing  COT 
TRREOE  did  not  have:  EiNX,  TRIPWIRE,  or  EMIO. 

2.  The  TRRLOE  database  was  provided  by  the  NRL  server.  Each  EAIRGAME  operational 
node  will  have  its  own  server  and  database. 

3.  Eor  TRRLOE:  MAGNET,  Global  Trader,  Port  and  Coastal  Surveillanee,  and  AIS  data 
were  available  only  through  CMA,  sueh  that  only  the  limited  data  from  them  that  was 
ineluded  in  the  CMA  data  build  were  available.  PASTC2AP  had  only  Volpe  AIS  data 
available. 
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4.  The  TRRLOE  FASTC2AP  was  able  to  obtain  data  from  only  Volpe  AIS.  FAIRGAME 
will  have  all  of  the  AIS  feeds. 

5.  The  TRRFOE  laboratory  CMA  workstations,  on  laptop  computers,  crashed  frequently 
during  the  morning  of  the  first  day.  This  problem  was  eliminated  by  installing  revised 
video  drivers  (from  the  Dell  website),  and  invoking  increased  system  cache  memory  to  be 
used  for  Java  applets. 

6.  During  FAIRGAME,  operators  will  utilize  Spiral-1  technologies  as  a  suite  of  available 
capabilities,  using  the  strengths  of  each  as  they  see  fit.  It  was  not  possible  to  determine 
which  systems  were  the  most  efficient  to  use,  or  preferred  by  operators. 

7.  Because  of  database  differences,  there  was  limited  ability  to  test  use  of  different  systems 
in  conjunction  with  each  other  to  accomplish  a  task.  For  instance,  participants  could  not 
locate  a  ship  on  Google  Earth  then  view  it  on  CMA  to  acquire  details  and  assess  further. 

A  ship  located  in  one  system  would  often  not  be  found  in  another. 

8.  Database  restrictions  meant  that  TRRFOE  events  had  to  be  configured  to  qualifying  ships 
that  were  identified  pre-event.  FAIRGAME  will  identify  ships  in  real-time. 

9.  Alerting  and  collaboration  were  not  available  on  CMA. 


Operational  Units: 

•  FAIRGAME  will  utilize  operational  units  at  various  organizations:  MIFCPAC,  NMIC, 
etc. 

•  TRRLOE  was  a  lab  test,  with  all  operators  in  a  single  room,  with  a  separate  White  Cell. 

Because  there  was  no  operational  higher  command,  the  White  Cell  had  to  play  that  role.  Also, 

handoff  and/or  collaboration  between  different  watch  floors  could  not  be  done. 

Operators: 

•  FAIRGAME  operators  will  be  watch  personnel  at  the  various  organizations. 

•  TRRLOE  used  an  IS2  Optical  Analyst  and  an  IS3  Strike  Analyst  operating  two  CMAs 
and  one  LT  Intelligence  Analyst  operating  the  other  two  systems  (FASTC2AP  and 
Google  Earth). 

The  TRRLOE  operators  had  a  great  deal  of  experience  and  had  only  positive  impact  on  the  test. 

Training: 

•  FAIRGAME  operational  personnel  will  have  extensive  training  by  an  official  mobile 
training  team. 

•  TRRLOE  operators  were  trained  on  the  day  before  the  operation  by  trainers-in-training. 
The  on-line  training  tools  were  not  used  for  TRRLOE  training. 

•  Additional  training,  as  required,  was  given  during  TRRLOE  execution. 
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Databases: 

•  FAIRGAME  will  utilize  real-time  data  provided  by  each  unit’s  servers. 

•  TRRLOE  used  mainly  historical  data  provided  by  the  NRL  CMA  server. 

Not  having  common  databases  for  the  three  systems  meant  that  they  could  often  not  be  used  in 
conjunction  with  each  other,  e.g.,  one  could  not  use  EASTC2AP  to  locate  a  ship,  or  receive  an 
alert,  then  use  CMA  to  gather  information  on  the  ship. 


2,3  Planned  TRRLOE  Events 

hollowing  are  the  events  that  were  planned  for  TRRLOE. 

Event-1  Obtain  information  on  named  ships 

WC  sends  an  RFI  requesting  all  available  information  on  two  named  ships.  One  of  the  ships  will 
have  complete  information  available,  the  second  limited  information.  The  named  ships  were: 
CMA-1  CMA-2 

Tocho  Maru  Shiraoi  Maru 

Polynesia  Sagittarius  header 

Expected  Operator  Actions  -  The  operator  will  search  for  information  for  each  ship.  RFI 
responses  for  both  ships  will  be  sent  to  the  WC.  For  the  ship  with  limited  information,  the 
operator  will  prepare  and  send  an  RFI  to  another  command  requesting  that  they  search  for  the 
missing  information.  This  secondary  RFI  will  also  be  sent  to  the  WC. 

Event-2  Correlate  ship  and  person-of-interest  information 

WC  sends  an  RFI  requesting  information  about  a  person  of  interest  who  is  reported  to  be  on  a 
named  ship.  WC  also  forwards  a  simulated  TRIPWIRE  tipper  concerning  the  person-of-interest. 
The  REI  will  request  information  about  the  person  and  the  ship. 

CMA-1  CMA-2 

Ship:  Shiraoi  Maru  Ship:  B  W  Prince 

Person:  Renato  Lumandas  Person:  Sang  Li  Tao 
Expected  Operator  Actions  -  The  operator  will  search  for  information  on  both  the  ship  and  the 
person.  It  will  be  found  that  the  person  is  not  on  the  named  ship.  The  operator  will  send  a 
response  to  the  WC  that  includes  information  on  both  ships  and  on  the  person. 

Event-3  Alert  of  ships  crossing  AOR  boundary 

WC  sends  an  RFI  requesting  that  ships  transiting  the  Suez  Canal  within  the  last  24  hours  or  that 
will  transit  within  the  next  24  hours  be  identified.  Information  requested  is  ship  identity, 
registry,  cargo,  and  whether  approaching  or  leaving  the  canal. 

Expected  Operator  Actions  -  The  operator  will  establish  a  geographic  boundary  that  will  be 
used  to  identify  ships  within  their  area  of  interest.  Ships  that  qualify  will  be  identified,  their 
requested  information  searched  for  and  obtained,  and  RFI  response  prepared  and  sent  to  the  WC. 

Event-4  VOI  Addition,  Sorting  and  tracking  named  vessels 

WC  names  three  vessels,  requesting  that  tracking  of  them  be  established.  One  will  be  moving 
toward  the  NAVCENT/PACFLT  boundary,  the  other  two  will  be  non-existent. 
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CMA-1&2 

Ship-1:  Catheryn  Maru  Ship-2:  Patrick  Angel  Ship-3:  YM  Inception 
Expected  Operator  Actions  -  Three  new  vessels  are  added  to  the  VOI  watch  list.  Both 
operators  loeate  the  existing  ship.  Operator  in  whose  AOR  ship  is  loeated  will  establish  traeking 
and  report  loeation.  Operators  collaborate  prior  to  handoff.  Ship  handoff  and  second  operator 
establishes  traeking  and  reports  location.  Operators  prepare  and  forward  RFI  for  ships  that 
weren’t  located. 

Event-5  Vessels  in  Proximity  of  Named  Ship 

WC  issues  an  RFI  to  identify  and  report  on  vessels  that  have  been  in  20nm  proximity  to  a  named 
ship,  within  200  nm  of  the  US  West  Coast. 

CMA- 1 :  Symphony  I 
CMA-2:  Shiraoi  Maru 

Expected  Operator  Actions  -  Each  operator  establishes  their  geographic  boundaries,  loeate 
ships  within  the  boundary.  CMA,  Google  Earth,  and  EASTC2AP  operators  collaborate  on 
loeating  ships.  A  report  will  be  sent  to  WC  providing  identities  of  qualifying  ships. 

Event-6  Identify  ships  within  a  geographic  area  that  are  carrying  hazardous  cargo 

WC  send  an  REI  requesting  identity  of  ships  earrying  hazardous  materials,  within  the  past 
month,  specify  a  geographic  area  and  request  identity  of  ships  and  all  available  information. 
Geographie  Area  CMA-1:  500nm  off  US  coast,  Eos  Angeles  to  San  Erancisco 
CMA-2:  200nm  around  Singapore 

Expected  Operator  Actions  -  Each  operator  establish  their  geographic  boundaries,  utilize 
advaneed  seareh  to  identify  ship  carrying  hazardous  cargo.  A  report  will  be  sent  to  WC 
providing  identities  of  qualifying  ships. 

Event-8  VOI  information  and  its  provenance 

WC  request  all  available  information  on  a  named  ship,  sourees  of  the  information,  and 
assessment  of  information  quality. 

CMA-1  ship:  BW  Prinee  CMA-2:  Toeho  Maru 

Expected  Operator  Actions  -  Search  for  the  ship’s  information,  record  the  information  and  its 
source,  apply  judgment  on  the  validity  of  the  information  and  report  results  to  WC. 

Event-9  Setting  up  alerts 

Repeat  the  Event-6  request  with  the  addition  that  automated  alerts  be  set  up  for  ships  meeting  the 
criteria. 

Criteria:  see  Event-6 

Expected  Operator  Actions  -  Establish  the  automated  alerts,  no  reporting  expeeted. 
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3.0  Test  Conduct  Philosophy  and  Execution 

The  primary  purpose  of  TRRLOE  was  to  reduee  risk  for  the  FAIRGAME  MDA  Spiral- 1  systems 
assessment.  This  meant  that  the  operators  and  the  systems  they  used  had  to  exereise  the  EG 
events  in  sueh  a  way  as  to  uneover  any  problems  that  might  be  eneountered  in  EG.  The  normal 
preeautions  one  takes  for  an  assessment  event,  most  importantly  to  ensure  reality  of  the 
operational  environment,  were  not  germane.  Completeness  of  risk  testing  was  of  primary 
importanee.  Thus,  the  RR  tests  foeused  on  determining  requirements  for,  and  impediments  to, 
operators,  systems,  and  databases  to  exeeute  the  EG  events. 


3.1  Test  Conduct 

Four  prineipal  eireumstanees  dietated  RR  eonduet: 

•  Only  three  of  the  Spiral- 1  systems,  and  no  base-line  systems,  were  available. 

•  Standard  Spiral- 1  training  for  the  operators  was  not  available. 

•  The  test  was  eondueted  in  a  laboratory. 

•  Historieal  databases  had  to  be  used. 

Seetion  2  deseribes  modifieations  to  the  test  events  that  were  due  to  these  eireumstanees.  This 
seetion  deseribes  how  they  affeeted  test  eonduet. 

Operational  Organization  Structure  -  The  operators  did  not  play  as  an  organization  with  an 
AOR.  Problems  were  mostly  in  the  form  of  general  RFIs  that  were  posed  to  the  two  CMA 
operators.  The  FASTC2AP/Google  Earth  operator  worked  the  problems  as  those  systems 
allowed. 

Real-Time  Training  -  Suffieient  training  for  the  operators  to  beeome  fully  eompetent  on  their 
systems  before  RR  start  was  not  available.  It  was  neeessary  to  allow  some  limited  assistanee 
during  the  tests  so  that  the  operators  eould  use  the  systems  eorreetly  and  effieiently. 

White  Cell  Operation  -  The  White  Cell  initiated  all  events  by  sending  an  RFI  to  the  operators. 
An  operator  would  provide  information  and  perhaps  ask  for  olarifieation.  Even  when  no 
olarifieation  was  requested,  the  White  Cell  would  often  ask  for  additional  information.  The 
White  Cell  was  not  hands-off,  rather  he  interaeted  with  the  operators  in  sueh  a  way  as  to  ensure 
they  fully  exereised  their  system’s  eapabilities.  This  methodology  was  in  eonformanee  with 
normal  MIFCPAC  operations. 

Communications  -  All  eommunieations  between  the  White  Cell  and  operators  were  via  a  Spark 
IM  elient  installed  on  a  laboratory  XMPP  server.  During  events  when  the  operators  needed  to 
eollaborate  it  was  done  faee-to-faee  sinee  the  eollaboration  eapability  in  CMA  was  not  available. 

Observer  Conduct  -  Observers  of  tests/experiments  normally  remain  in  the  baekground  as  mueh 
as  possible  so  as  not  to  distort  results.  A  risk-reduetion  event  is  a  speeial  ease  sinee  its  primary 
purpose  is  to  identify  risk  to  another  event,  not  to  gather  assessment  data.  It  was  important  for 
RR  that  all  events  be  exeeuted  and,  in  order  for  that  to  oeeur,  some  interaetion  between 
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operators,  trainers,  and  observers  was  required.  This  interaction  didn’t  affect  assessment  of 
system  performance,  human-system  interaction,  and  operator  acceptance/evaluation  of  the 
systems. 


System  Utilization  -  White  Cell  RFIs  were  sent  to  the  two  CMA  operators.  Since  they  had  no 
suite  of  systems  to  use,  they  were  limited  to  acquiring  all  information  from  CMA.  No  directions 
were  sent  directly  to  the  FASTC2AP/Google  Earth  operator;  he  worked  in  conjunction  with  one 
of  the  CMA  operators  as  was  allowed  by  their  respective  databases  (when  the  same  ships  were 
available).  Almost  all  FASTC2AP  use  entailed  testing  of  the  various  pre-set  alert  capabilities. 

MDA  Processes  Quality  and  Execution  Time  -  The  above  noted  behaviors  are  compatible  with 
a  risk  reduction  event  but  do  introduce  artificiality.  Capturing  data  that  will  lead  to 
determination  of  the  quality  of  operational  activities  and  tasks,  and  the  time  required  for  their 
execution,  requires  a  realistic  operational  environment.  In  a  lab  event  of  this  type,  only  times 
associated  with  acquiring  data  from  the  systems  could  be  determined. 

There  were  3  observers  for  this  event:  Doug  MacKinnon  (NPS),  Sue  Hutchins  (NPS),  and  David 
Rousseau  (SSC-SD).  Gordon  Schacher  (NPS)  performed  exercise  control  duties  for  this  event. 


3.2  TRRLOE  Real-Time  Event  Modifications 

Some  of  the  TRRLOE  events  were  modified  during  test  execution  because  of  unforeseen 
circumstances.  The  following  list  those  modifications  and  their  reasons. 

Event- 1:  No  modifications. 

Event-2:  No  modifications. 

Event-3:  The  event  had  already  been  modified  from  FAIRGAME’s  original  intent  to  observe 
ships  transiting  from  one  AOR  to  another.  Because  of  the  limited  ship  database,  it  was  not 
possible  to  identify  candidate  ships.  Thus,  the  event  was  modified  to  identify  ships  transiting 
into  and  from  the  Suez  Canal.  It  was  further  simplified  during  the  event  to  identify  only  ships 
that  are  in  the  canal. 

Event-4:  CMA  collaboration  capabilities  were  not  available  so  the  handoff  portion  of  the  event 
could  not  be  done. 

Event-5:  20  nm  proximity  was  used  to  locate  ships  of  interest.  The  search  was  then  narrowed  to 
identify  ships  with  which  there  could  have  been  a  rendezvous. 

Event-6:  No  change. 

Event-7:  FAIRGAME  EV07  was  not  exercised  because  there  was  no  container  information  in 
the  TRRLOE  laboratory  system. 
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Event- 8:  No  Change. 


Event-9:  Not  exereised  beeause  the  laboratory  CMA  system  did  not  have  authority  to  ereate 
alerts.  The  struetures  and  eapabilities  of  EASTC2AP  alerts  were  tested,  but  not  as  an  event. 

3,3  Conduct  Impact  on  Results 

The  following  are  the  factors  introduced  by  RR  conduct  and  their  impact  on  risk  reduction 
results. 

Operators  and  Training  -  The  operators  were  experienced  in  intelligence  watch  operations  but 
had  minimal  training  on  the  systems.  This  allowed  determination  of  training  requirements  but 
there  was  no  ability  to  identify  risk  due  to  operator  effects  such  as  their  system-use  preference. 

System  Availability  -  The  systems  that  were  available  had  a  complete  test  for  usability  and 
operator  acceptance.  There  was  no  ability  to  test  impact  that  operator  preference  for  using 
Spiral- 1  vs  using  existing  systems  might  have  on  EAIRGAME. 

Database  -  The  non-conventional  databases  that  were  available  in  the  laboratory  was  a 
fortuitous  circumstance.  It  allowed  identification  of  database  problems  that  could  impact  EG. 

Organization  Effects  -  Realistic  operational  environments  were  not  utilized  so  no  risk  associated 
with  organizations  nor  their  processes  could  be  tested. 


As  noted  previously  in  this  report,  RR  was  not  conducted  as  an  assessment  test.  Even  so, 
assessment  results  can  be  extracted.  It  will  not  be  possible  to  assess  operational  performance, 
such  as  task  execution  times  because  of  the  unrealistic  environment  and  test  procedures.  It  will 
be  possible  to  determine  usability  of  the  systems,  operator  acceptance  and  their  evaluation  of 
system  impact  on  their  activities,  and  training  needs.  Assessment  results  that  could  be 
determined  are  presented  in  Section  6. 
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4.0  TRRLOE  Observer  and  Survey  Results 


These  results  are  based  on  extraetions  of  data  from: 

•  Observer  logs 

•  Hot-wash-ups  that  were  eondueted  with  all  personnel  at  the  end  of  eaeh  day 

•  Surveys  that  were  eompleted  by  the  operators 

The  first  two  sourees  are  summarized  in  Seetion  4.1. 

These  results  are  summaries  obtained  direetly  from  the  data  sourees,  prior  to  detailed  analysis. 
Analysis  results  are  presented  in  Seetions  5  and  6,  risk  reduetion  and  MDA  analysis  results, 
respeetively. 


4,1  Observer  Results 

The  following  results  are  for  CMA,  exeept  where  noted. 

Information  Acquisition  Times  -  Even  though  the  systems  were  not  used  in  an  operational 
environment,  the  amount  of  time  required  to  obtain  information  using  them  was  representative. 
Very  little  time  was  required.  E.g., 

•  1  min  to  determine  when  ship  was  built  after  White  Cell  request. 

•  7  min  to  determine  person-of-interest  was  not  on  reported  ship. 

•  2  min  to  identify  eargo  on  a  named  ship. 

•  10  min  to  loeate  a  ship  and  identify  its  origin  and  arrival  loeations. 

•  6  min  to  loeate  a  ship,  details  on  type  and  erew. 

It  is  possible  that  these  times  are  biased  to  shorter  time  beeause  historieal  data  was  used  for 
whieh  the  needed  data  was  known  to  be  present.  Even  so,  these  information  aequisition  times 
represent  an  improvement  over  existing  proeedures. 

Operator  Performance/! raining  -  Operator  training  on  the  operations  they  were  asked  to 
perform  with  the  systems  was  minimal.  Within  this  eontext,  the  following  observations  ean  be 
made: 

•  They  developed  faeility  with  the  systems  in  a  short  period  of  time,  usually  after  about  14 
hour  of  working  with  a  funetionality. 

•  How  the  system  applied  to,  or  eould  be  used  for,  the  jobs  with  whieh  they  were  familiar 
was  elear.  Diffieulties  they  had  were  largely  the  result  of  system  malfunetion  (due  to  lab 
environment). 

•  There  were  instanees  where  an  operator  searehed  for  a  funetion  or  eapability  CMA  did 
not  have.  This  eould  be  a  training  issue  and/or  represent  funetionalities  that  should  be 
added. 

•  It  is  elear  that  two  or  three  days  of  foeused  training  on  the  Spiral- 1  suite  will  result  in 
fully  eompetent  operators. 
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Human-System  Interaction  -  Note  that  the  following  HSI  observations  are  based  on  operators 
who  were  not  fully  trained  on  the  systems.  More  eomplete,  operationally-based,  training  would 
alleviate  some  of  the  diffieulties  eneountered.  These  observations  apply  to  CMA  unless 
otherwise  noted. 

•  The  layout  of  advaneed  seareh  is  eumbersome  (layout  and  seareh  parameters  disappear 
when  new  seareh  parameters  are  entered). 

•  The  seareh  information  used  when  initiating  a  search  cannot  be  seen  when  scrolling  down 
in  the  display.  This  can  result  in  losing  track  of  overall  search  context.  Operators 
requested  a  horizontal  rather  than  vertical  layout  of  information. 

•  Having  political  and  AOR  boundaries  would  aid  the  operator  in  setting  up  search  areas. 

•  Searching  for  and  acquiring  information  can  involve  many  information  elements.  It 
would  be  useful  to  have  a  pre-structured  notepad  where  the  operator  could  enter 
information  already  obtained  and  build  an  information  picture. 

•  Operators  tend  to  develop  personalized  methods  for  information  searches.  It  would  be 
useful  to  have  a  means  for  capturing  operator  searches  for  reuse.  This  would  be 
especially  useful  for  categories  in  advanced  search. 

•  Add  ‘cargo’  to  pull  down  search  menu. 

•  FASTC2AP:  The  structure  of  the  format  used  for  setting  up  alerts  is  unclear  and  difficult 
to  use. 

•  FASTC2AP:  Add  capability  to  make  a  polygon,  a  common  shape  required  to  conduct  a 
search  in  the  real  world. 

•  Add  feature  to  specify  a  point  and  a  specified  radius  to  establish  a  search  area. 

•  Add  feature  to  be  able  to  enter  a  coordinate  and  then  be  able  to  set  up  a  search  area 
around  it. 

•  Need  better  color  coding  along  left-hand  side  for  icons;  make  meaning  of  color  explicit. 

•  Need  better  understanding  of  what  pre-built  searches  do;  user  thought  they  were  going  to 
provide  different  information. 

•  Need  ability  to  save  an  area  when  drawing  a  box/polygon  with  a  specific  name. 

•  Change  hyper-graph:  provide  more  detailed  information  (e.g.,  vessel  name)  and  make  it 
easier  to  read  information 

•  Include  the  ability  to  draw  a  line  to  be  able  to  search  any  track  crossing  a  line  rather  than 
a  box. 

System  Performance  -  Most  system  problems  encountered  were  due  to  the  lab  environment  and 
do  not  qualify  as  Spiral- 1  system  results.  The  following  results  are  inherent  to  the  systems,  not 
the  lab  environment.  These  results  are  derived  from  limited  tests  of  the  systems  and  have  not 
been  validated. 

•  Partial  match  did  not  work  as  anticipated  when  searching  for  a  ship  name. 

•  Operators  could  not  zoom  in  and  draw  a  small  geographical  box  around  a  VOI. 

•  Advanced  search  is  difficult  to  use.  Specifications  can  be  lost  when  adding  to  search. 

•  Basic  and  advanced  search  need  more  defined  categories,  such  as  “hazmat”  for  cargo. 

•  It  would  be  advantageous  to  be  able  to  use  distances  when  setting  up  searches. 

•  FASTC2AP:  Three  of  the  alerts  (Course  Proximity,  Vessel  Proximity,  and  Next 
Waypoint  Feasible,  require  the  SCONUM.  Matching  of  ship  name,  position  and 
SCONUM  becomes  classified.  Since  FASTC2AP  will  be  fielded  on  CENTRIX,  perhaps 


16 


the  ship’s  Lloyd’s  or  International  Maritime  Organization  (IMO)  number  could  be 
substituted  for  the  currently  required  SCONUM. 

•  FASTC2AP:  Boolean  criteria  alerts  were  difficult  to  set  up  and  all  conditions  may  not  be 
available. 

•  FASTCAP:  Needs  capability  to  refine  searches  more  by  user:  current  searches  are  too 
broad. 

•  FASTCAP:  Understanding  the  results  of  agents  was  difficult 

•  FASTCAP:  Could  not  designate  an  agent  to  find  ships  that  came  within  200nm  proximity 
of  the  west  coast. 

•  FASTCAP:  Flexible  template  relations  were  difficult  to  understand. 

•  FASTCAP:  Interface  was  not  intuitive  to  use. 

•  Need  separate  category  to  rate  the  age  of  the  data  -  an  important  criteria  on  which 
operational  tasking  is  built  on.  (There  is  a  quality  rating,  but  time  lateness  of  data  should 
be  a  separate  category.) 

•  Make  the  factors  that  go  into  “quality”  rating  transparent  to  the  user. 

•  Need  a  clear  indicator  to  inform  user  when  any  database  is  down. 

•  GOOGLE  EARTH:  Eatest  position  was  easy  to  get,  but  the  Volpe  AIS  data  was  limited 
and  not  useful  for  in-depth  analysis. 

•  GOOGEE  EARTH:  Could  search  for  the  latest  position,  but  not  for  track  data. 


4,2  Survey  Results 

Surveys  were  administered  for  each  of  the  technologies  involved  in  the  TRRLOE  in  addition  to 
demographics  and  training  surveys.  System  specific  survey  questions  were  developed  in  addition 
to  the  general  survey  questions  that  appear  in  all  three  surveys.  Surveys  used  for  TRRLOE  MDA 
technologies  are  included  in  Appendix  A.  The  purpose  of  the  surveys  was  to  gather  feedback 
from  the  users  involved  in  the  exercise  to  identify  strengths  and  weaknesses  of  the  MDA 
technologies.  Topics  included  training  effectiveness,  system  usability,  suggestions  for 
improvement,  and  features  participants  found  particularly  useful. 

Demographics 

Three  users  participated  in  the  TRRLOE;  two  participants  used  the  CMA  technology  and  one 
used  EASTC2AP  and  Google  Earth  systems  throughout  the  experiment.  Participants  included  an 
IS-3  Strike  Analyst,  an  IS-2  Optical  Analyst,  and  a  ET  Intelligence  Analyst,  currently  attending 
the  Naval  Postgraduate  School.  The  average  service  time  for  the  participants  was  about  five 
years  and  their  level  of  experience  in  their  current  positions  varied  between  intermediate  and 
advanced. 

Baseline  Training  Survey  Results 

All  three  system  operators  had  no  prior  knowledge  of  Comprehensive  Maritime  Awareness 
(CMA)  or  PASTC2AP  technologies,  or  their  capabilities,  prior  to  the  training.  All  users  received 
an  instructional  brief,  and  depending  on  the  participant,  some  received  hands-on  training,  a  quick 
reference  guide,  and  trainer  interaction.  Two  users  felt  that  the  training  materials  were  easy  to 
understand  given  the  amount  of  time  allowed  to  complete  the  tasking,  and  the  other  felt  that  the 
material  was  not  easy  to  sift  through.  Participants  thought  the  preparation,  instruction,  and 


17 


training  provided  during  TRRLOE  for  CMA  and  FASTC2AP  was  fairly  productive,  all  users  felt 
that  there  should  have  been  more  time  allotted  in  order  to  adequately  understand  the  processes 
and  systems  as  a  whole,  in  addition  to  working  one-on-one  with  subject  matter  experts. 

Regarding  Google  Earth,  all  users  had  prior  knowledge  of  its  capabilities.  All  users  thought  the 
instructional  briefs  and  on-line  tutorials  were  easy  to  understand  and  also  agreed  the  training  and 
instruction  received  prepared  them  for  the  tasks  performed  during  TRRLOE. 

Comprehensive  Maritime  Awareness  (CMA) 

Regarding  the  training  materials  provided,  one  user  felt  that  for  the  limited  time  allotted,  the 
materials  made  it  fairly  easy  to  complete  the  tasking  while  the  other  user  felt  that  the  training 
materials  were  not  easy  enough  to  navigate  through  and  suggested  an  online  version  to  improve 
the  method  of  searching.  One  participant  felt  strongly  about  allowing  more  time  during  the 
exercise  in  order  to  adequately  comprehend  the  material,  while  the  other  user’s  concern  was  to 
provide  more  one-on-one  time  with  operators  and  subject-matter  experts.  One  participant 
suggested  that  the  system  should  have  been  fully  developed  beforehand  to  enable  the  participants 
to  know  what  to  focus  on  during  the  training  and  get  the  most  out  of  the  exercise  along  with 
more  scenarios  in  order  to  practice  on  prior  to  the  test  evaluations.  Although  complications  were 
experienced  during  the  exercise,  overall,  the  users  felt  satisfied  with  the  training  they  received. 

FASTC2AP 

The  FASTC2AP  operator  had  approximately  one  hour  of  training  on  the  system  prior  to  the 
exercise,  where  he  was  first  introduced  to  the  technology.  Two  operators  received  training  on  the 
system  in  the  form  of  an  instructional  brief  which  they  both  agreed  was  easy  to  understand. 
Participants  agreed  the  training  was  effective,  but  it  could  have  been  longer,  and  suggested  that 
more  scenarios  be  added  to  the  exercise  in  order  to  practice  more  before  evaluations,  and  also 
requested  more  one-on-one  time  with  operators. 

GOOGLE  EARTH 

For  the  Google  Earth  application,  all  system  operators  had  prior  knowledge  of  the  technology. 
Participants  received  an  on-line  tutorial  in  addition  to  an  instructional  briefing  that  they  felt  was 
easy  to  understand  and  use;  the  training  was  adequate  for  them  to  effectively  utilize  the 
technology.  Due  to  the  familiarity  of  the  application,  the  users  were  satisfied  with  the  training 
they  received  and  felt  that  they  could  use  Google  Earth’s  technology  effectively  by  the  end  of  the 
exercise. 


Comprehensive  Maritime  Awareness  (CMA)  Survey 

Usability.  Both  CMA  operators  thought  the  technology  was  easy  to  learn  and  use,  and  the 
training  produced  basic  knowledge  of  the  system  and  its  background.  Both  CMA  users  strongly 
agreed  it  was  easy  to  develop  and  maintain  situation  awareness  on  a  vessel  of  interest  (VOI), 
especially  once  provided  with  a  watch  list  that  includes  the  names  of  the  vessels.  CMA  operators 
emphasized  that  it  was  extremely  easy  to  find  relevant  information  on  a  VOI  as  long  as  one  knew 
the  name  of  the  vessel;  otherwise  it  would  be  difficult  due  to  the  hyper  graph. 
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It  was  easy  for  CM  A  users  to  assoeiate  tipper/  Intel  information  with  the  correct  VOL  CM  A 
showed  recent  alerts  on  the  vessel  and  made  it  easy  to  see  the  tippers.  Operators  maintained 
contact  with  the  white  cell  (representing  higher  command)  on  chat.  On  day  one,  users  found  it 
difficult  to  cross  reference  tracks  being  used  by  operators  on  other  systems  due  to  the  difficulty 
of  looking  at  two  units  and  not  being  able  to  compare  them.  However,  on  day  two,  CMA  users 
agreed  that  the  use  of  either  FASTC2AP  or  Google  Earth  made  it  much  easier  to  cross  reference 
and  pull  up  the  necessary  information  to  collaborate. 

Suggestions  for  Improvement  and  Useful  Features.  Both  CMA  users  experienced  difficulties 
using  the  alerting  system  as  well  as  the  hyper  graph.  Participants  noted  that  when  searching  on 
CMA  using  a  partial  search,  the  system  did  not  always  pull  up  all  tracks  containing  a  portion  of 
the  name.  One  user  recommended  adding  a  feature  to  provide  the  capability  to  use  range  and 
bearing  from  a  point  in  a  circle  to  form  a  search  area.  This  user  also  suggested  including  the 
ability  to  use  a  line  or  border  to  search  for  any  track  crossing  a  line  rather  than  a  box  as  is  done  in 
the  current  system. 

Another  user  noted  that  when  attempting  to  save  a  box  or  area,  the  program  would  not  warn  the 
user  when  an  existing  file  had  the  same  file  name,  causing  confusion  when  referring  back  to  a 
basic  search  in  the  pull  down  menu,  and  finding  two  files  with  the  same  name.  The  user  also 
suggested  that  the  system  should  include  a  search  pull  down  in  the  metadata  section  that  allows 
for  the  search  of  cargo,  making  it  easier  to  narrow  down  the  hits.  CMA  users  stated  that  the 
scenarios  were  realistic  enough,  and  foresee  the  CMA  system  becoming  a  very  useful  tool  for  all 
AORs  once  incorporated.  Participants  thought  conducting  analysis  on  a  VOI  was  easier  and 
much  faster  when  using  CMA  than  with  their  previous  system. 

FASTC2AP 

Usability.  The  participant  felt  that  the  program  was  difficult  to  learn  resulting  in  the  user  having 
to  try  the  agents  multiple  times  in  order  to  understand  what  each  was  providing  in  the  way  of 
results.  On  the  second  day  the  FASTC2AP  operator  stated  it  was  fairly  easy  to  create  agents, 
however  understanding  the  results  was  still  difficult.  This  user  was  not  able  to  utilize  the  online 
aid,  nor  did  he  know  what  the  software  “wizards”  were.  The  FASTC2AP  operator  thought  it  was 
easy  to  compose  agents,  although  the  system  could  not  provide  the  details  required  for  some  of 
the  RFIs.  This  user  found  it  very  useful  to  build  alerts  in  order  to  fill  RFIs  through  the 
FASTC2AP  alerting  system.  Since  this  exercise  only  looked  at  snapshot  type  problems,  it  was 
hard  to  maintain  situation  awareness  on  a  VOI  given  that  they  did  not  track  a  VOI  over  a  period 
of  time.  When  collaborating  with  other  operators  during  the  scenario,  the  operator  was  able  to 
cross  reference  some  tracks,  however  CMA  and  FASTC2AP  had  access  to  different  data,  making 
it  somewhat  difficult. 

Suggestions  for  Improvement  and  Useful  Features.  At  one  point  this  user  wanted  to  find  ships 
that  came  within  200  NM  proximity  of  the  west  coast,  and  he  was  not  able  to  designate  an  agent 
to  carry  out  the  task,  suggesting  that  this  capability  should  be  added.  This  user  also  added  that 
the  flexible  template’s  relations  were  difficult  to  understand  and  suggested  that  the  graphical  user 
interface  (GUI)  be  made  more  intuitive.  Easily,  this  participant  suggested  removing  the  ship 
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control  number  as  a  required  field  when  using  it  on  an  unelassified  network.  Overall,  the  user 
viewed  FASTC2AP  as  an  effeetive  VOI  analysis  tool  with  the  eorreet  data  to  support  it. 


GOOGLE  EARTH 

Usability.  The  user  felt  that  this  applieation  was  easy  to  learn,  and  use,  finding  no  diffieulty  in 
using  all  of  the  features  included  in  Google  Earth.  This  user  stated  the  system  was  a  quiek 
geospatial  reference  for  developing  awareness  on  a  VOI  as  well  as  finding  information  on  them. 
Condueting  analysis  on  a  VOI  was  easy,  speeifieally  for  aequiring  their  last  position;  however 
the  VOLPE  AIS  data  was  limited,  and  not  useful  for  in-depth  analysis  or  for  providing  traeking 
data.  There  was  no  embedded  eollaboration  tool  ineluded,  thus  it  was  not  possible  to  send 
information  from  Google  Earth  to  other  systems.  However  it  was  possible  to  look  up  vessels  in 
other  systems  making  it  easier  to  eross  referenee  traeks  and  information  on  VOI  with  other 
operators. 

Suggestions  for  Improvement  and  Useful  Features.  The  user  found  it  useful  as  a  platform  for 
displaying  data  and  not  as  a  eollaboration  tool.  Other  than  the  missing  information  due  to  the 
differenee  in  databases  and  the  restrietions  of  the  applieation,  the  user  found  the  system  to  be 
very  helpful  throughout  the  exereise.  The  feature  this  participant  found  most  helpful  was  the 
“find”  option,  providing  an  easy  way  to  see  if  a  VOI  was  in  the  data  feed  as  well  as  finding  basie 
information  on  the  VOI. 
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5.0  FAIRGAME  Risk  Reduction  Results 


5,1  Identified  Risks  with  Mitigations 

1 .  Databases  for  CMA  (Comprehensive  Maritime  Awareness)  are  developed  by  NRL  and 
exported  to  the  various  operational  nodes.  These  databases  are  AOR  (Area  of 
Responsibility)  specific,  so  different  AORs  may  not  have  the  same  ship  tracks.  This 
could  impact  track  sharing,  collaboration,  and  handoff  events. 

Mitigation;  Databases  for  CMA  will  need  to  be  verified  prior  to  FAIRGAME 
execution  to  ensure  that  same  tracks  can  be  viewed  and  manipulated.  This  will 
ensure  that  collaboration  and  VOI  hand-off  can  occur. 

2.  NRL  databases  have  been  undergoing  modifications  and  sometimes  are  re -based  with 
software  version  installations. 

Mitigation:  We  had  them  “lock  down”  the  database  for  TRRLOE  so  that  it  would 
be  stable.  (Lock  down  in  the  sense  that  there  is  not  a  restart  that  deletes  past 
data.)  This  should  be  done  for  EAIRGAME  so  that  available  tracks  remain 
through  both  the  planning  and  execution  phases  of  EAIRGAME. 

3.  Ships  utilized  for  TRRLOE  events  were  identified  before  the  event.  It  was  found  that 
there  were  very  few  qualifying  ships  in  the  laboratory  database.  It  took  two  days  of  work 
to  obtain  appropriate  ships.  This  may  have  been  due  to  the  lab  database  being  only  about 
one  week  old  and  may  not  be  a  factor  for  EAIRGAME.  Regardless,  identifying 
appropriate  ships  was  time  consuming.  The  EAIRGAME  plan  is  to  identify  ships  to 
“play”  in  the  events  in  real-time.  This  could  be  difficult  to  do. 

Mitigation;  It  is  recommended  that  test  runs  be  made  before  the  event  to 
determine  if  real-time  ship  selection  will  work. 

4.  The  TRRLOE  systems  had  problems  with  data  ingests  to  CMA  from  both  Global  Trader 
and  MAGNET.  We  do  not  know  if  this  is  only  a  laboratory  systems  problem  or  extends 
to  fielded  systems  as  well. 

Mitigation:  EAIRGAME  planners  should  do  an  inventory  of  exactly  what  data  is 
being  provided  by  and  to  the  Spiral- 1  system  nodes  to  ensure  that  the  various 
systems  will  have  required  data  available  for  each  event. 

5.  Setting  up  alerts  in  CMA  requires  that  the  operator  have  a  higher  level  of  permission  than 
NRL  gave  to  the  TRRLOE  Lab  140.  The  event  requiring  CMA  alerts  could  not  be  run. 

Mitigation;  The  permission  levels  of  the  operational  nodes  should  be  checked  to 
ensure  that  they  are  high  enough  to  accomplish  all  EAIRGAME  test  events  - 
especially  alert  building  events. 
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6.  FAIRGAME  survey  questions  address  performanee  of  Spiral-1  teehnologies  as-a-whole. 
The  test  is  set  up  so  that  operators  ean  use  available  systems  as  they  wish;  they  will  use 
both  Spiral- 1  and  existing  systems.  It  is  possible  that  a  partieular  Spiral- 1  system  may 
have  no  or  little  use  while  another  may  have  significant  use. 

Mitigation;  In  this  situation,  assessing  the  systems  as-a-whole  can  produce 
skewed  results.  It  is  suggested  that  surveys  be  modified  so  that  individual 
systems  are  addressed. 

7.  The  scenario  used  for  TRRLOE,  which  was  based  on  the  planned  scenario  for 
EAIRGAME,  was  CMA-centric. 

Mitigation:  Ensure  scenario  events  are  included  that  will  make  use  of  the  specific 
features  provided  by  other  MDA  technologies. 

8.  Very  little  hands-on  training  was  provided. 

Mitigation:  Ensure  training  includes  operationally-oriented  examples  of  how  to 
accomplish  tasks  with  hands-on  practice. 
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6.0  MDA  Assessment  Results 


As  noted  above,  beeause  of  the  risk  reduction  format  for  TRRLOE,  there  was  limited  ability  to 
obtain  assessment  results.  Even  so,  worthwhile  assessment  results  were  obtained. 

1.  Significant  Reduction  in  Time  Required  to  Access  Information.  The  TRRLOE 
operators  were  two  experienced  Information  Specialists  (IS)  and  one  Intelligence  Officer. 
IS  comments  revealed  that  use  of  CMA  would  significantly  improve  their  ability  to 
access  required  information.  One  IS  commented,  “Without  CMA  it  can  take  anywhere 
between  several  hours  to  two  days  to  build  a  picture  on  a  VOL  Eirst  time  used  CMA  was 
able  to  pull  up  the  same  info  in  a  matter  of  minutes:  cargo,  who  owns  vessel,  ship 
characteristics,  and  they  agree  across  different  sources.”  The  significant  reduction  in 
access  and  fusion  time  could  result  in  providing  more  time  for  higher-level  analysis  and 
for  monitoring  more  vessels. 

2.  System  Operators  Found  MDA  Systems  Very  Useful.  Even  though  use  of  the  four 
Spiral- 1  systems  in  conjunction  could  only  be  done  in  a  limited  way,  the  operators 
commented  that  these  tools  would  be  a  useful  for  analysts,  again  speeding  up  their 
processes.  CMA  operators  agreed  it  was  easy  to  develop  and  maintain  situation 
awareness  on  a  VOI,  especially  if  one  knows  the  name  of  the  vessel.  “All  the  info  is  at 
your  fingertips.”  It  is  a  bit  harder  if  there  are  a  large  number  of  vessels  that  requires 
searching  information  on  each  one.  CMA  operators  strongly  agreed  it  was  easy  to  find 
relevant  information  on  a  VOI  (although  they  did  not  like  the  hyper-graph  format  for 
displaying  information).  EASTCAP  user  thought  alerts  he  built  to  fill  REIs  were  very 
useful. 

3.  Change  Alerts  so  SCONUM  is  Not  Required.  EASTC2AP  has  a  number  of  pre-defined 
alert  functions.  Their  use  requires  an  ONI-assigned  SCONUM,  which  when  combined 
with  a  ship’s  name  and  its  position  becomes  classified.  The  SCONUM  also  may  not  be 
available  to  coalition  forces.  This  makes  these  alerts  of  limited  use  and  could  cause 
future  security  concerns.  It  is  recommended  that  the  alerts  be  rewritten  to  require  ship 
names  only  or  the  Lloyd’s  identification  number. 

Operational  use  of  MDA  Spiral- 1  systems  will  be  in  conjunction  with  existing  systems.  It  is 
expected  that  operators  will  develop  preferences  for  how  they  use  the  systems,  ways  that  are 
natural  in  concord  with  their  experience  and  training.  MDA  information  capture  and  assessment 
should  determine  these  use  methods  as  they  evolve  as  an  aid  in  developing  CONOPS,  TIP,  and 
training.  Such  determinations  were  not  made  during  TRRLOE. 
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Appendices 


A.  Surveys 

Two  types  of  information  were  obtained  from  the  partieipants.  The  first  provides  data  eolleetion 
via  a  User  Log  Sheet,  shown  for  CMA  only  below.  Its  purpose  is  to  log  the  information 
assoeiated  with  vessels  on  whieh  they  are  working.  The  seeond  instrument  is  a  survey  to  obtain 
system  information  and  its  use. 

The  following  are  the  log  sheets  and  surveys  for  CMA  only.  Systems  logs  and  surveys  are  eaeh 
tailored  to  speeifie  systems,  yet  these  CMA  instruments  are  suffieient  for  this  illustration.  Mueh 
of  the  spaee  provided  for  information/answers  are  removed  to  make  them  shorter  for  this 
doeument. 


CMA  User  LOG  SHEET 

Please  complete  the  table  below  as  you  engage  in  the  scenario.  List  all  Vessels  of  Interest 
(VOIs)  you  process  and  other  tracks  that  you  process. 


VOI 

What 
caused 
you  to 
notice 
this 

Vessel? 

Critical  Info  Obtained  on 
VOI 

Alerts  on  VOIs  Received 

Time 

Noticed 

Start  Time 
of 

Processing 

End  Time 
of 

Processing 

Total  Time 
for 

Processing 
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CMA  User  Survey 

Comprehensive  Maritime  Awareness 

Please  select  one  response  to  each  of  the  following  items.  Use  the  space  below 
each  item  to  furnish  additional  information  about  your  responses. 

1.  I  have  used  CMA  previously. 

_ Yes  _ No 

If  yes,  please  identify  what  exactly: _ 

2.  I  had  prior  knowledge  of  this  technology’s  capabilities. 

_ Yes  _ No 

If  yes,  please  explain: _ 

3.  CMA  was  easy  to  learn. 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 


4.  CMA  was  easy  to  use. 

Strongly  Disagree 

Neutral 

Agree 

Strongly 

Not 

Disagree 

Agree 

Applicable 

□  □ 

□ 

□ 

□ 

□ 

Please  explain: 


5.  The  online  help  was  useful. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 


6.  It  was  easy  to  use  all  the  features  included  in  CMA. 

Strongly 

Disagree 

Neutral 

Agree 

Strongly 

Not 

Disagree 

Agree 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain  those  features  that  were  not  easy  to  use: _ 

7.  CMA  made  it  easier  to  do  my  job  than  with  the  current/  previous  system  I  was/  am  using. 
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Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 

8.  CMA  made  it  faster  to  do  mv  job  than  with  the  current/  previous  svstem  I  am/  was  using. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 

9.  I  was  able  to  develop  awareness  of  VOIs  faster  with  CMA. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 

10.  It  was  easy  to  find  information  on  a  vessel  of  interest. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 

11.  The  alerting  system  in  CMA  was  easy  to  understand. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

12.  The  alerting  svstem  in  CMA  was  useful  and  relevant. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

13.  It  was  easy  to  develop  and  maintain  situation  awareness  on  a  vessel  of  interest. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

14.  It  was  easy  to  find  relevant  information  on  a  vessel  of  interest. 
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Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

15.  It  was  easy  to  associate  tipper/  Intel  information  with  the  correct  vessel  of  interest. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

16.  It  was  easy  to  conduct  analysis  on  a  vessel  of  interest. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

17.  It  was  easy  to  collaborate  with  others  on  a  vessel  of  interest. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 

18.  It  was  easy  to  search  and  track  a  vessel  of  interest. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

19.  I  was  able  to  process  and  maintain  situation  awareness  on  more  VOIs  than  with  my  previous 
system. 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

20.  i  was  abie  to  process  and  maintain  situation  awareness  on  more  than  one  VOi  at  a  time  more 
easiiy  than  with  my  previous  system. 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 
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21.  It  was  easy  to  cross  reference  tracks  being  processed  by  operators  using  other  systems. 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

Please  explain: 

□ 

□ 

□ 

□ 

□ 

22.  Were  there  actions  you  wanted  to  perform  with  the  system  that  you  did  not  understand? 

Yes  _  No  _ 

Please  explain:  _ 

23.  Were  there  actions  you  wanted  to  perform  with  the  system  that  were  not  available? 

Yes  _  No  _ 

Please  explain:  _ 

24.  Was  there  any  conflicting  information  when  you  collaborated  with  users  on  other  systems? 

Yes  _  No  _ 

Please  explain:  _ 

25.  Was  there  any  information  confusion  when  you  collaborated  with  users  on  other  systems? 

Yes  _  No  _ 

Please  explain:  _ 


26.  I  could  collaborate  easily  about  the  same  information  with  operators  using  other  systems. 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Not 

Applicable 

□ 

□ 

□ 

□ 

□ 

□ 

Please  explain: 

27.  It  was  easy  to  cross  reference  information  on  VOIs  with  operators  using  other  systems. 

Yes  _  No  _ 

Please  explain:  _ 

28.  I  could  access  and  use  the  online  help. 

Yes  _  No  _ 

Please  explain:  _ 

29.  What  features  did  you  find  to  be  helpful  in  performing  your  tasks? 

Please  explain  how/  why  each  feature  listed  was  helpful  (be  as  specific  as  possible). 

Feature  Used  Why  was  it  helpful? 


30.  Were  there  any  features  in  CMA  that  you  found  difficult  to  use? 

Please  explain  how/  why  each  feature  listed  was  helpful  (be  as  specific  as  possible). 

Feature  Used  Why  was  it  difficult  to  use? 
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31.  I  view  this  as  an  effective  collaboration  tool. 

Yes  _  No  _ 

Please  explain:  _ 

32.  What  changes,  if  any,  would  you  recommend  to  improve  CMA? 


33.  Was  there  any  conflicting  information  when  you  collaborated  with  other  system  operators  on 
a  VOI,  and  what  was  it? 

Yes  No 

Please  explain:  _ 

34,  Any  other  comments?  _ 
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B.  Observer  Logs 

Observer  logs  are  used  to  reeord  time-stamped  events.  They  provide  data  whieh  are  analyzed  to 
produce  both  quantitative  and  qualitative  information.  Below  are  the  directions  for  the  logs 
followed  by  the  spreadsheet  form. 

TRRLOE  Observer  Log  -  Directions 

The  following  explains  the  purpose  for  and  use  of  the  columns  in  the  log. 

MEL  #  Enter  the  #  for  the  event  that  is  underway  when  the  observation  is  made. 

Time  Enter  the  time  of  the  observation  (not  the  time  the  observation  is  recorded). 

Observation  This  is  the  data.  It  is  what  we  will  analyze.  Enter  anything  you  feel  is  pertinent 
for  evaluating  quality  of  performance  or  system.  Especially  important  is  context,  such  things  as 
the  operator  is  confused.  Also  record  such  things  as  the  number  of  steps  it  takes  to  perform  a 
task.  Be  sure  to  record  the  start  and  end  times  for  tasks.  Use  more  than  one  line  for  a  single 
observation,  if  needed. 

Check-Boxes  Check  all  boxes  that  are  appropriate  for  your  observation.  These  checks  will  be 
used  during  analysis  to  sort  the  data.  If  no  box  is  checked,  the  analyst  will  have  to  figure  out  to 
what  the  data  applies  (not  good). 

•  CMA  GE  ECP  Shorthand  for  the  three  systems  being  used.  Check  only  one  for  a 
data  entry. 

•  System  Performance,  Usability,  Efficiency,  Clarity  These  are  for  system  evaluation. 

•  Information  Availability,  Quality  Judgments  about  the  information  received,  or 
produced  by  the  operator. 

•  Operator  Capability,  SA,  Knowledge  Judgments  about  the  operator. 

•  Timeliness,  Accuracy  These  can  apply  to  any  of  the  observations. 

Check-Box  Rules  The  following  apply  to  what  can  be  checked  for  a  single  observation. 

Only  one  system  -  Observations  for  more  than  one  system  require  more  than  one  entry. 

Timeliness  and  Accuracy  -  Can  be  checked  whenever  appropriate,  no  restrictions. 

System,  Information,  Observer  -  Only  one  of  the  three  categories  can  be  checked  for  an 
observation.  However,  more  than  one  block  in  a  category  can  be  checked. 

Archiving  Instructions; 

1 .  Open  the  spreadsheet  from  the  “Eogs  Eolder”  in  thumb-drive  you  have  been  provided. 

2.  Record  your  information  from  the  day’s  observations  on  the  spreadsheet. 
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3.  File  the  spreadsheet  using  your  initials  and  the  observation  day, 

E.g.,  GS-Observer  Log-1. 

You  will  are  hive  your  “Observer  Evaluation  of  Activities”  survey  form  in  the  same  way. 

E.g.,  GS-Activities  Survey-1 

When  first  archiving,  also  rename  your  folder  with  your  initials,  e.g.,  GS-Logs  Eolder. 


TRRLOE  Observer  Log 

The  following  is  a  compressed  form  of  the  log  spreadsheet. 


MEL 

#  Time 


iTimeliness 


C.  Hot  Wash-Up  Comments 

User  Comments 

1 .  CMA  is  a  good  tool  beeause  it  reduces  the  time  to  get  information. 

2.  CMA  is  a  valuable  asset  due  to  the  broad  amount  of  information  it  provides  in  a  short 
time.  All  information  comes  up  in  one  location;  don’t  have  to  look  at  so  many  windows 
as  is  currently  done. 

3.  Would  have  been  good  if  I  had  a  better  understanding  up  front  of  what  it  was  supposed  to 
do,  i.e.,  is  it  replacing  GCCS,  or  what? 

4.  Not  getting  a  COP  with  MDA;  are  getting  a  set  of  tools. 

5.  Need  more  operationally -based  training.  Scenarios  had  MDA  focused  questions  -  use 
those  as  hands  on  training. 

6.  FastC2AP  and  Google  Earth  are  good  tools.  FastC2AP  needs  capability  to  refine  searches 
more  by  user;  current  searches  are  too  broad.  Need  to  be  able  to  narrow  searches. 

7.  Need  better  understanding  of  what  the  pre-built  searches  do.  Need  to  be  very  clear  on 
what  each  agent  does.  When  tried  to  use  them,  thought  it  was  going  to  provide  different 
information. 

8.  Takes  so  long  to  do  a  search  of  Sealink.  With  CMA  you  have  one  place  to  go  which 
saves  a  lot  of  time. 

9.  Time  savings:  Without  CMA  it  can  take  two  days  to  build  a  picture  on  a  VOI.  First  time 
used  CMA  was  able  to  pull  up  the  same  info  in  a  matter  of  minutes:  cargo,  who  owns 
vessel,  ship  characteristics,  and  they  agree  across  different  sources. 

10.  Some  difficulty  accomplishing  tasks,  e.g.,  search  specific  500nm  area.  Huge  region 
comes  up  with  so  many  vessels.  MIFCPAC  subject-matter  expert  said  it’s  a  realistic 
request.  But  petty  officer  thought  it  would  be  better  to  make  it  a  more  focused  request. 
(Indicated  that  is  how  she  receives  requests  in  explot,  i.e.,  more  specific.) 

11.  Usability  issue:  Search  Metadata  -  cargo,  30  days  in  X  area.  Would  be  easier  to  have  pull 
down  menu  include  cargo. 

12.  When  draw  a  box  and  tried  to  save  as  Singapore  for  basic  search  area,  wound  up  with  two 
boxes  being  saved  (because  had  done  this  step  twice).  Add  feature  similar  to  MS  Office 
where  it  asks  ‘Are  you  sure?’  Or  gives  an  opportunity  to  name  to  ‘save  as’. 

13.  FastC2AP  flexible  template  page:  have  to  name  vessel  looking  at,  have  to  formulate 

relationship,  characteristics  of  search: _ , _ .  And  check  add.  (i.e.,  complicated) 

14.  Proximity  Vessel  -  another  set  of  input,  have  to  enter  time,  range,  refines  search  faster. 
GUI  not  intuitive  to  use. 

15.  Only  get  a  rectangular  box:  can’t  get  a  polygon,  which  is  a  pretty  common  thing  that 
would  be  needed  to  do  a  search  in  real  world.  This  capability  needs  to  be  included  in 
FastC2AP. 

16.  We  only  looked  at  snapshot  type  problems  and  did  not  look  at  VOIs  over  a  period  of  time 
(in  response  to  question  asking  whether  it  is  easy  to  maintain  awareness  on  a  VOI). 

17.  FASTCAP  and  CMA  have  access  to  different  data  making  it  difficult  to  cross  reference 
tracks. 
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This  page  intentionally  left  blank. 
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D.  Master  Events  List 


The  following  are  the  TRRLOE  events  as  designed  prior  to  execution.  The  table  contains  the 
FAIRGAME  event  number  in  column  1  and  the  Master  Event  List  number  in  column  2.  The  last 
two  columns  indicate  actions  to  be  taken  by  the  White  Cell  and  the  operators. 

Note  that  not  all  FAIRGAME  events  are  executed.  Also,  there  is  an  extra  event,  EVxx,  inserted. 


TRRLOE  Master  Events  List 


FG# 

MEL# 

Description 

WC 

Oper 

EVOl 

RRl.l 

White  Cell  issue  RFI  referencing  a  particular  vessel. 

W 

RR1.2 

Acquire  all  available  information  about  the  ship. 

0 

RR1.3 

Locate  the  ship. 

0 

RR1.4 

Search  for  an  alert  for  the  ship. 

0 

RR1.5 

Analyze  information  obtained  for  completeness. 

0 

RR1.6 

Issue  needed  RFI  to  other  organizations. 

0 

RR1.7 

Prepare  RFI  response. 

0 

RR1.8 

Distribute  RFI  response. 

0 

RR1.9 

White  Cell  evaluate  RFI  response  and  forwarded  RFI. 

W 

EV02 

RR2.1 

White  Cell  issue  RFI  referencing  a  person-of-interest  and  their  ship. 

W 

RR2.2 

WC  sends  pseudo-TRIPWIRE  report  about  the  person. 

W 

RR2.3 

Search  for  information  on  the  reported  person. 

0 

RR2.4 

Acquire  all  available  information  about  the  ship. 

0 

RR2.5 

Acquire  information  about  the  person's  ship. 

0 

RR2.6 

Locate  both  ships. 

0 

RR2.7 

Prepare  RFI  response. 

0 

RR2.8 

Distribute  RFI  response. 

0 

RR2.9 

White  Cell  evaluate  RFI  response. 

W 

0 

EV03 

RR3.1 

Request  alerts  for  ship  movements  across  AOR  boundary. 

W 

RR3.2 

Build  alert. 

RR3.3 

Establish  collaboration  between  operators. 

0 

RR3.4 

Observe  alert  and  inform  collaboration  partner. 

0 

RR3.5 

Confirm  ship  locations. 

0 

RR3.6 

Collaborate  on  ship  locations. 

0 

RR3.7 

Track  ships. 

0 

RR3.8 

Collaborate  on  reporting  ships  at  boundary. 

0 

RR3.9 

Report  ships  approaching  /  crossing  boundary. 

0 

EV04 

RR4.1 

WC  sends  3  NMIC  named  VOI,  requests  tracking  and  reporing. 

W 

RR4.2 

Vessels  are  added  to  the  watch  floor  VOI  list. 

0 

RR4.3 

Establish  collaboration  between  operators. 

0 

RR4.4 

Operator  collaboration  on  VOI  addition. 

0 

RR4.5 

Establish  cross-AOR  coordination. 

0 

RR4.6 

Track  ship. 

0 
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RR4.7  Initial  tracking  operator  report  position. 

0 

RR4.8  Handoff  ship. 

0 

RR4.9  Second  tracking  operator  report  location. 

0 

RR4.10  Issue  RFI  for  non-located  ships. 

0 

RR4.11  White  Cell  evaluate  forwarded  RFI. 

W 

EVxx  RR5.1  WC  sends  RFI  to  ID  and  report  vessels  in  20nm  proximity  to 
named  ship. 

W 

RR5.2  Proximity  alert  set  up. 

0 

RR5.3  Vessels  triggering  alert  identified. 

0 

RR5.4  Proximity  vessels  reported  to  the  WC. 

0 

EV06  RR6.1  Send  RFI  defining  geographic  area  and  ship  attributes. 

W 

RR6.2  Search  for  ship  information. 

0 

RR6.3  Determine  ship  positions,  correlate  with  area-of-interest. 

0 

RR6.4  Fuse  ship  information  and  ID  ships  that  meet  criteria. 

0 

RR6.5  Prepare  list  of  conforming  ships. 

0 

RR6.6  Forward  list  of  conforming  ships  to  WC. 

0 

EV08  RR8.1  Specify  ship  and  request  ship  information  and  its  provenance. 

W 

RR8.2  Locate  ship  information. 

0 

RR8.3  Prepare  report  on  ship  information,  information  sources,  and  quality. 

0 

RR8.4  Distribute  ship  information. 

0 

EV09  RR9.1  Specify  and  forward  ship  attributes  and  geographic  areas. 

W 

RR9.2  Develop  alerts  for  areas-of-interest. 

0 

RR9.3  Develop  alert  for  ship  that  is  known  to  be  within  Area  of  Interest. 

0 

RR9.4  Determine  location  of  ships  that  meet  attributes. 

0 

RR9.5  Observe  correct  alert  functioning. 

0 
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E.  White  Cell  Directions 


White  Cell  (WC)  directions  will  be  provided  to  2  CMA  operators  and  another  person  operating 
both  FASTC2AP  and  Google  Earth.  The  following  directions  specify  tasks  for  CMA-1  and 
CMA-2.  The  third  operator  will  execute  tasks  as  appropriate  for  the  capabilities  of  the  systems. 

Operator  output  products  will  be  sent  to  the  WC.  The  WC  will  evaluate  these  products  for  their 
quality  using  a  provided  questionnaire. 

Event-1  Obtain  information  on  named  ships. 

WC  send  an  RFI  requesting  all  available  information  on  two  named  ships.  One  of  the  ships  will 
have  complete  information  available,  the  second  limited  information.  The  named  ships  are: 
CMA-1  CMA-2 

Tocho  Maru  Shiraoi  Maru 

Polynesia  Sagittarius  Leader 

Expected  Operator  Actions  -  The  operator  will  search  for  information  for  each  ship.  RFI 
responses  for  both  ships  will  be  sent  to  the  WC.  For  the  ship  with  limited  information,  the 
operator  will  prepare  and  send  an  RFI  to  another  command  requesting  that  they  search  for  the 
missing  information.  This  secondary  RFI  will  also  be  sent  to  the  WC. 

Event-2  Correlate  ship  and  person-of-interest  information. 

WC  send  an  RFI  requesting  information  about  a  person  of  interest  who  is  reported  to  be  on  a 
named  ship.  WC  also  forward  a  simulated  TRIPWIRE  tipper  concerning  the  person-of-interest. 
The  RFI  will  request  information  about  the  person  and  the  ship. 

CMA-1  CMA-2 

Ship:  Shiraoi  Maru  Ship:  B  W  Prince 

Person:  Renato  Lumandas  Person:  Sang  Li  Tao 

Expected  Operator  Actions  -  The  operator  will  search  for  information  on  both  the  ship  and  the 
person.  It  will  be  found  that  the  person  is  not  on  the  named  ship.  The  operator  will  send  a 
response  to  the  WC  that  includes  information  on  both  ships  and  on  the  person. 

Event-3  Alert  of  ships  crossing  AOR  boundary. 

WC  sends  an  RFI  requesting  that  ships  transiting  the  Suez  Canal  within  the  last  24  hours  or  that 
will  transit  within  the  next  24  hours  be  identified.  Information  requested  is  ship  identity, 
registry,  cargo,  and  whether  approaching  or  leaving  the  canal. 

Expected  Operator  Actions  -  The  operator  will  establish  a  geographic  boundary  that  will  be  used 
to  identify  ships  within  their  area  of  interest.  Ships  that  qualify  will  be  identified,  their  requested 
information  searched  for  and  obtained,  and  RFI  response  prepared  and  sent  to  the  WC. 


Event-4  VOI  Addition,  Sorting  and  tracking  named  vessels. 

WC  names  three  vessels,  requesting  that  tracking  of  them  be  established.  One  will  be  moving 
toward  the  NAVCENT/PACFLT  boundary,  the  other  two  non-existent. 
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CMA-1&2 

Ship-1;  Catheryn  Maru  Ship-2:  Patrick  Angel 


Ship-3:  YM  Inception 


Expected  Operator  Actions  -  Three  new  vessels  are  added  to  the  VOI  watch  list.  Both  operators 
loeate  the  existing  ship.  Operator  in  whose  AOR  ship  is  loeated  establish  tracking  and  report 
loeation.  Operators  collaborate  prior  to  handoff.  Ship  handoff  and  seeond  operator  establishes 
tracking  and  reports  loeation.  Operators  prepare  and  forward  RFI  for  ships  that  weren’t  located. 


Event-5  Vessels  in  Proximity  of  Named  Ship 

WC  issues  an  RFI  to  identify  and  report  on  vessels  that  have  been  in  20nm  proximity  to  a  named 
ship,  within  200  nm  of  the  US  West  Coast. 

CMA- 1 :  Symphony  I 
CMA-2:  Shiraoi  Maru 


Event-6  Identify  ships  within  a  geographic  area  that  are  carrying  hazardous  cargo. 

WC  send  an  RFI  requesting  identity  of  ships  earrying  hazardous  materials,  within  the  past 
month,  speeify  a  geographie  area  and  request  identity  of  ships  and  all  available  information, 

Geographic  Area  CMA-1:  500nm  off  US  coast,  Los  Angeles  to  San  Franeisco 
CMA-2;  200nm  around  Singapore 

Expected  Operator  Actions  -  Eaeh  operator  establish  their  geographic  boundaries,  utilize 
advanced  seareh  to  identify  ship  carrying  hazardous  cargo.  A  report  will  be  sent  to  WC 
providing  identities  of  qualifying  ships. 


Event-8  VOI  information  and  its  provenance. 

WC  request  all  available  information  on  a  named  ship,  sourees  of  the  information,  and 
assessment  of  information  quality. 

CMA-1  ship;  BW  Prince  CMA-2;  Tocho  Maru 

Expected  Operator  Actions  -  Seareh  for  the  ship’s  information,  reeord  the  information  and  its 
source,  apply  judgment  on  the  validity  of  the  information  and  report  results  to  WC. 


Event-9  Setting  up  alerts. 

Repeat  the  Event-6  request  with  the  addition  that  automated  alerts  be  set  up  for  ships  meeting  the 
criteria. 

Criteria:  see  Event-6 

Expected  Operator  Actions  -  Establish  the  automated  alerts,  no  reporting  expected. 
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